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FOREWORD 


This  semi-annual  technical  report  presents  the  accomplishments  during 
the  period  of  1  June  1973  to  30  November  1973.  The  principal  investigator 
is  Dr.  D.  C.  Lai  and  co-principal  investigator  is  Dr.  T.  Kailath.  Since 
the  inception  of  this  project,  it  has  been  the  result  of  the  collaborative 
efforts  of  many  individuals.  In  particular,  Dr.  J.  E.  Anliker  has  spent 

endless  hours  on  this  project.  Most  of  the  staff  members  contributed  in 
the  writing  of  this  report. 
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Biocybernetic  Factors  in  Human  Perception  and  Memory 

SUMMARY 

This  project  is  concerned  with  the  application  of  biocybernetic 
concepts  to  the  problem  of  expanding  human  memory.  In  particular,  the 
goal  of  this  research  is  enhancement  of  visual  imagery  as  far  as  possible 
in  the  direction  of  photographic  memory.  Scanpaths  of  the  eye  during 
inspection  of  visual  targets  are  treated  as  indicators  of  the  brain's 
strategy  for  the  intake  of  visual  information.  This  research  will  determine 
the  features  that  differentiate  scanpaths  associated  with  superior  imagery 
from  scanpaths  associated  with  inferior  imagery.  Similarly,  the 
tachistoscopic  exposure  of  the  visual  material  is  responsible  for  a  number 
of  adjustments  in  the  brain's  waking  rhythm;  the  electroencephalographic 
features  correlated  with  superior  imagery  will  be  differentiated  from 
those  correlated  with  inferior  imagery.  A  computerized  biocybernetic 
scheme  will  be  implemented  in  an  attempt  to  generate  image  enhancement 
and  to  train  the  individual  to  exert  greater  voluntary  control  over  his 
own  imagery.  In  this  progress  report,  we  give  an  account  of  our 
accomplishments  made  in  this  reporting  period. 

Work  in  eye-movement  measurement  and  tracking  has  been  extensive. 

New  techniques  for  eye-movement  prediction  and  special  software  for 
visual  scanpaths  study  have  been  developed  and  implemented.  We  also 
describe  an  EEG  signal  model  which  prescribes  the  stimulus-response 
relationship.  The  information  gained  in  this  model  will  elucidate  the 
behavior  of  the  alpha  frequency  and  phase,  and  thus  lead  to  a  more  practical 
brain  state  estimator  and  predictor  for  our  use  in  the  memory  studies 
of  this  project . 
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.  .  1.  inL 


The  implications  of  this  project  for  the  Department  of  Defense  arise 
from  the  possibility  of  devising  new  and  unusual  techniques  that  will 
permit  the  development  of  stronger  imagery  in  men  with  normal  memories. 
Since  it  is  definitely  an  asset  for  the  military  man  to  have  the  ability 
to  absorb  information  rapidly  and  to  retain  it  intact  for  long  periods 
of  time,  this  project  will  attempt  to  intensify  the  post-stimulus  imagery 
as  far  as  possible  in  the  direction  of  photographic  memory— ,md  to  search 
out  those  features  of  successful  recall  strategies  so  as  to  enhance  the 
individual’s  power  of  redintegration. 
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L .  INTRODUCTION 


The  primary  concerr  of  this  research  project  is  the  development 
of  biocybernetic  concepts  and  techniques  required  for  the  analysis  and 
development  of  skills  useful  for  the  manipulation  and  control  of  memory 
functions,  particularly  those  related  to  the  more  concrete  ("eidetic") 
images.  In  particular,  we  are  concentrating  on  the  problem  of  achieving 
biocybernetic  expansion  of  human  visual  memory  through  the  use  of  real¬ 
time  computerized  monitoring  and  feedback  of  cues  that  serve  as  keys  to 
memory  encoding  and  retrieval.  This  approach  represents  a  more  or  less 
logical  extension  of  the  considerable  evidence  that  the  human  nervous 
svstem  depends  heavily  upon  spatial  and  temporal  cues  both  in  the  encoding 
and  decoding  of  memories,  especially  sensory  images. 

Our  plan  is  to  develop  and  implement  techniques  for  real-time 
monitoring  and  prediction  of  central  nervous  activities  through  the  electro- 
encephalographic  signals  and  through  eye-movement  signals.  We  shall  use 
this  information  to  arrange  extraordinary  coincidences  between  various 
brain  states,  eye  positions,  and  the  delivery  of  visual  stimulation. 

The  visual  stimuli  are  to  be  presented  either  on  the  screen  of  a  storage 
scope  or  in  complex  tachistoscopic  batteries  for  either  monopic  or 
dichopic  viewing.  The  eye  position  and  eye  movements  will  be  measured 
by  a  unified  system  so  as  to  obtain  their  optical  and  electrophysiological 
estimators.  Combining  these  techniques  (in  order  to  monitor  eye  pointing 
both  when  eyelids  are  open  and  when  closed)  and  using  a  feedback  scheme 
to  close  the  control  loop,  we  expect  to  obtain  greater  control  of  image 
persistence  and  image  dissipation. 

Our  work  has  been  progressing  in  the  direction  to  achieve  the 
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following  goals: 


(1)  Determine  the  temporal  and  spatial  cues  that  serve  as  keys 
to  memory  encoding  and  retrieval  via  the  tracing  of  brain  states 
and  eye  movement. 

(2)  Design  and  implement  real-time  estimation  and  prediction  algorithms 
for  monitoring  brain  states  and  eye  pointing. 

(3)  Use  computer  memory  to  supplement  and  strengthen  the  aspects 
of  human  memory  that  ordinarily  result  in  image  dissipation.  We 
assume  that  a  superior  pattern  of  visual  inspection  (i.e.,  one  that 
results  in  superior  memory)  is  more  consistent  and  also  less  probable 
of  natural  occurrence  than  a  visual  inspection  strategy  that  is  less 
consistent.  Through  the  use  of  computer  guidance  (feedback)  we 

will  steer  the  subject  toward  improved  encoding  and  improved  decoding 
strategies . 

In  this  report,  we  describe  our  accomplishments  in  developing  and 
implementing  those  concepts  and  techniques  which  are  essential  in  achieving 
our  goals. 

The  heart  for  the  implementation  of  our  schemes  is  the  PDP-15  computer 
system  which  was  acquired  in  the  last  contractual  year.  This  system 
has  been  fully  utilized  and  expanded  by  the  addition  of  16K  core  memory 
and  a  cartridge  disk  system.  This  addition  enhances  the  capabilities  of 
the  system  for  real-time  data  acquisition  and  processing  which  are  pre¬ 
requisites  for  the  realization  of  our  schemes.  The  updating  and  refine¬ 
ment  of  the  system  has  been  our  continuing  effort.  However,  we  are  not 


going  to  report  here  our  accomplishments  in  this  endeavor. 

In  this  period,  our  major  effort  was  devoted  to  eye-movement  measure¬ 
ment  and  tracking.  Progress  has  been  made  in  the  monitoring  of  eye 
position  and  predictions  of  fixation  points.  This  is  described  in  Section 
III.  In  the  next  section,  we  present  new  techniques  in  calibrating  our 
eye-movement  monitor  system  and  in  correcting  the  distortion  due  to  the 
system.  This  will  enable  us  to  obtain  more  accurate  eye-movement  data. 

We  have  also  completed  a  computer  program  dubbed  "SEER"  that  was  designed 
to  open  up  a  new  avenue  for  the  study  of  visual  scanpaths.  We  anticipate 
using  this  software  heavily  during  the  next  quarter  and  producing  many 
interesting  results  in  our  study  of  eye  movement  and  eye  position.  The 
"SEER"  program  is  described  in  Section  IV.  In  Section  V,  we  present  an 
EEC  signal  model  which  relates  the  visual  stimuli  and  the  EEG  signal. 

The  understanding  of  this  stimulus-response  relationship  will  help  us  in 
the  determination  of  the  temporal  features  that  play  prominent  roles  in 
the  encoding  and  decoding  of  memories.  We  make  some  concluding  remarks 
and  describe  future  work  in  the  last  section. 
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II.  EYE-MOVEMENT  MEASUREMENT: 

CALIBRATION  AND  DISTORTION  CORRECTION 

The  success  of  our  project  hinges  on  the  determination  of  both  the 
spatial  and  temporal  cues  that  serve  as  keys  to  memory  encoding  and  recall. 
For  the  determination  of  the  spatial  cues,  it  is  imperative  that  we  should 
have  an  accurate  technique  for  the  measurement  of  eye  movement  and  eye 
position.  At  present,  the  only  instrument  for  measuring  eye  movement 
available  to  us  is  the  Eye-Movement  Monitor  (Type  SG)  manufactured  by 
Biometrics,  Inc.  The  Biometric  Eye-Movement  Monitor  provides  a  simple 
and  relatively  easy-to-use  method  for  measurement  of  horizontal  and 
vertical  movements  of  the  eye.  In  the  horizontal  direction,  the  method 
is  based  on  the  measurement  of  the  amount  of  infra-red  light  reflected 
from  the  junction  of  the  iris  and  the  sclera.  In  the  vertical  direction, 
the  eyelid-sclera  junction  is  used.  This  method  of  measurement,  un¬ 
fortunately,  yields  unreproducible  and  highly  distorted  results.  Several 
sources  of  noise  exist: 

(1)  Movement  of  the  head  or  the  rpectacles  on  which  the  sensors 
are  mounted  results  in  translation  of  the  X,Y  coordinate  space. 

(2)  The  horizontal  signal  changes  as  the  vertical  position  of 

the  eye  is  varied.  This  problem  is  referred  to  as  "vertical  cross¬ 
talk." 

(3)  The  vertical  signal  suffers  from  an  "eyelid  hysteresis" 
phenomenon,  that  is,  the  eyelid  does  not  always  return  to  the  same 
relative  position  after  vertical  movement  or  blinks. 
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(4)  Nonlinearities  are  introduced  in  both  the  X  and  Y  signals  due 
to  changes  in  the  adjustment  of  the  apparatus  and  through  differences 
in  the  characteristics  of  the  eyes  of  the  subjects. 

As  reported  in  the  last  annual  technical  report,  we  have  made  extensive 
modifications  on  this  unit  to  suit  our  needs.  Since  then,  we  have  improved 
our  calibration  technique  and  implemented  new  techniques  for  distortion 
correction.  A  subroutine  "iBALL"  has  been  written  for  the  purpose  of 
measuring  the  reproducibility  of  the  measurement  of  eye  fixations.  In 
other  words,  it  will  give  some  reasonable  bounds  on  the  errors  which  might 
be  expected  and  summarize  certain  statistical  properties  such  as  the  means 
and  variances  of  the  measured  eye  fixations.  This  information  will,  in 
turn,  aid  the  adjustment  of  the  positions  of  the  sensor-source  groups 
which  are  mounted  on  the  Biometric  eye-glass  frame.  We  intend  to  estimate 
the  ranges  of  errors  of  the  Biometric  Eye-Movement  Monitor  by  conducting 
repetitive  experiments  using  "IBALL"  program.  We  shall  describe  the 
operations  and  our  results. 

An  operational  flowchart  for  this  program  is  shown  in  Figure  II. 1. 

The  experiment  can  be  initiated  by  following  a  dialog  with  "IBALL"  as 
shown  in  figure  II. 2.  A  calibration  matrix  as  shown  in  Figure  II. 3  is 
used  to  aid  in  the  adjustment  of  the  sensor-source  units.  This  matrix 
is  presented  to  the  subject  on  a  storage  scope  point-at-a-time  in  random 
sequence  and  hence,  the  distance  and  direction  traveled  by  the  eyes  are 
randomized.  The  subject  is  requested  to  press  a  button  after  he  has 
fixated  on  the  prompted  point.  At  this  time,  ten  pairs  of  the  horizontal 
and  vertical  voltages  from  the  Biometric  unit  are  sampled  and  averaged. 

This  procedure  is  repeated  until  all  of  the  nine  points  in  the  matrix 


Figure  II. 1  Flowchart  of  IBALL 
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Figure  II. 2  Initial  Dialog  of  IBALL 


Figure  II. 3  Calibration  Matrix  for 
Biometrics  liye-Movement  Monitor 
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have  been  presented  and  thus  the  first  trial  is  completed.  Another  trial 
will  start  automatically .  We  show  two  typical  results  in  Figures  II. 4 
and  II. 5  accompanied  by  summaries  of  statistical  properties.  In  the 
figures,  the  circlos  on  the  pictures  are  the  positions  of  the  points  in 
the  calibration  matrix  shown  and  the  intersection  of  the  bars  of  each 


cross  indicates  the  mean  position  where  the  eyes  have  fixated.  The 

lengths  of  the  upright  and  horizontal  bars  are  proportional  to  the  standard 

deviations  in  the  vertical  and  horizontal  directions,  respectively,  of  the 

eye-fixation  points.  Ranges  of  horizontal  and  vertical  errors  obtained 

in  fifteen  experiments  are  shown  in  Figures  II. 6  and  11.7.  The  average 
o  o 

errors  are  0.7  and  1.2  for  horizontal  and  vertical  values,  respectively. 
The  average  error  ranges  are  0.4°  to  1.2°  and  0.4°  to  2.4°  for  horizontal 
and  vertical  error,  respectively. 

As  seen  from  Figures  11.4  and  II. 5,  the  mean  locations  of  the  eye- 
fixations  (as  indicated  by  the  crosses)  have  considerable  skewing  and 
distortion  caused  by  the  vertical  crosstalk  and  nonlinearities.  We  now 
describe  a  technique  which  attempts  to  correct  the  vertical  corsstalk 
and  nonlinearities  mentioned  above  .  The  technique  is  based  on  the 
construction  of  a  piecewise  linear  mapping  between  the  measurement  or 
response  space  (the  voltages  at  the  terminals  of  the  Biometrics  unit) 
and  the  stimulus  space  (points  in  the  visual  field  being  observed  by 
the  subject ) . 


If  we  let  (r  ,r  )  represent  the  measured  X,Y  voltage  values 


corresponding  to  some  stimulus  point  (s^s  ),  then  the  most  general  form 
of  a  linear  mapping  between  r  and  s  is  of  the  form 


s,  =  c  ,  r  +  c  r  +  c 
1  11  1  12  2  13 
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s  =  c  r+c  r+c 
2  21  1  22  2  23 

The  coefficients  c  are  defined  by  use  of  a  three  point  calibration  array 
which  empirically  determines  the  relationship  between  the  stimulus  values 
(s  ,s  )  and  the  response  values  (r  ,r  ). 

J.  ^  ±  £ 


Measured  value 


Stimulus  value 


(rla>r2.) 

corresponds  to 

(3la'S2a> 

<rlb’r2b) 

corresponds  to 

(Slb'S2b) 

<ric'r2c> 

corresponds  to 

<Slc’S2c> 

This  correspondence  allows  us  to  set  up  two  sets  of  three  simultaneous 
linear  equations,  the  first  of  which  is 

Sla  '  Cllrla  +c12r2a  +  °13 

Slb  cllrlb  +  c12r2b  +  c 13 

“lc  -  cllrlc  +  C12r2c  +  c13  . 

Solution  of  these  equations  yields  the  required  coefficients. 

In  practice,  one  such  linear  mapping  is  not  sufficient  to  correct 
for  distortioi  s  introduced  by  the  Biometrics  equipment.  The  problem, 
in  general,  is  to  find  the  minimum  number  of  stimulus  calibration  points 
(and  thus  the  number  of  mappings)  needed  to  reduce  the  nonlinear  distortion 
to  a  level  below  the  other  noise  sources  mentioned  above.  We  have  found 
that  a  calibration  matrix  of  five  to  nine  points  (four  to  eight  mappings) 
provides  adequate  linearization  for  our  experiments. 

A  program  has  been  written  which  computes  the  c  coefficients  for 
the  mappings.  It  consists  of  two  subroutine  calls  named  MAPSET  and 
MAPCNV.  The  visual  field  is  divided  into  three  to  eight  pie-shaped 
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sectors  around  a  pivot  point  in  the  center 


ui  uiu  visual  lie 


of  coefficients  is  computed  for  each  sector  by  MAPSET.  The  second  routine 
MAPCNV,  determines  which  sector  applies  to  a  given  measured  point  (r^r,) 
and  computes  the  corresponding  stimulus  value  (s  .s^). 

As  demonstration  of  the  effectiveness  of  this  distortion  correction 
program,  we  applied  this  technique  to  the  measured  eye-fixations  of 
subjects  7  and  5.  The  original  positions  of  the  eye  fixations  before 
correction  arc  shown  in  Figures  II.  5  and  II. 9  and  the  corrected  locations 
are  shown  In  Figures  II. 8  and  II, 10.  Judging  by  these  pictures,  we 
conclude  that  the  correction  by  the  use  of  this  computer  program  is 
quite  effective. 
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Figure  II. 9  Highly  Distorted  Before  Correction 
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Figure  1 1 . 10  Distortion  Corrected 
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III.  TRACKING  OF  EYE  MOVEMENT 


1 .  Introduction 

The  tracking  of  eye  movement  is  an  essential  part  of  our  research 
project.  In  order  to  feedback  the  spatial  cues  in  encoding  and  decoding 
of  memories,  we  have  to  track  constantly  the  movement  of  the  eyes. 
Basically,  for  the  kind  of  eye  movements  we  are  interested  in,  we  can 
classify  them  into  two  modes;  viz.,  fixation  and  saccades.  Thus,  for  the 
purpose  of  tracking  we  need  to  (a)  estimate  and  predict  the  duration  of 
a  fixation  when  in  that  mode,  i.e.,  to  predict  when  the  mode  will  change 
from  a  fixation  to  a  saccade  and  (b)  estimate  and  predict  the  duration, 
the  direction,  and  the  length  of  the  saccade  for  the  saccadic  movement 
itself,  i.e.,  to  predict  when  and  where  the  saccade  will  end. 

The  two  parts  are  interdependent.  Ultimately,  we  will  build  a  model 
which  will  take  care  of  both.  However,  more  insight  can  be  obtained  if 
we  deal  with  the  two  separately  and  then  combine  them  into  one  predictive 
model.  At  present,  we  have  looked  into  part  (L)  .  For  the  sake  of 
simplicity,  we  will  first  develop  a  deterministic  model  which  ignores 
the  problem  of  the  direction  and  the  stochastic  properties  of  the  saccade 
(assuming  it  to  be  a  straight  line)  and  gives  a  prediction  ol  the  final 
position  after  looking  at  the  onset  of  the  saccade.  The  estimation  of  the 
length  and  duration  of  the  saccade  is  the  more  difficult  part  of  the 
problem.  Some  empirical  observations  have  been  made  and  on  the  basis  of 
these,  we  have  developed  two  different  predictive  processes;  viz.,  the 
position-variance  method  and  the  velocity  method.  The  details  of  these 
methods  will  be  described  later.  The  realization  of  various  eye-movement 
tracking  techniques  requires  preprocessing  and  manipulation  of  the  raw 
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eye-movement  data.  To  this  end,  we  have  written  an  interactive  program 
for  eye-movement  analysis  (IPEMA). 

2.  An  Interactive  Program  for  Eye-Movement  Analysis  (IPEMA) 

This  program  combines  various  packages  for  analysis  of  eye  movements. 
It  interacts  with  the  user  through  the  DECwriter  or  the  VT05,  and  the 
data  switches  on  the  PDP-15  console.  The  eye-movement  data  is  read  off 
a  magnetic  tape.  The  data  is  stored  on  the  tape  in  a  particular  format  - 
files  consisting  c-I  records  of  63  data  samples  each.  Through  the  program, 
the  tape  can  be  positioned  at  any  particular  point  and  the  data  can  then 
be  processed. 

The  program  first  calls  an  initialization  subroutine  which  sets  the 
data  channels,  determines  if  the  plotting  device  is  the  CRT  or  the 
CalComp  plotter  and  initializes  the  device  and  positions  the  tape  to  the 
beginning  of  the  first  fire  of  the  specified  volume.  It  also  reads  in 
the  calibration  data  (standard  calibration  could  be  used  if  the  actual 
data  is  not  available)  and  sets  up  calibration  parameters. 

Now  the  main  program  prompts  the  user  with  a  "?."  Any  of  a  number 
of  commands  can  be  entered  by  the  user.  A  command  consists  of  five 
characters  followed  by  an  integer  number.  The  commands  essentially  fall 
into  three  classes: 

(i)  Commands  which  position  the  tape.  These  commands  can  move  the 
tape  to  the  beginning  of  a  particular  file  or  to  any  record  on  the 
file.  They  can  also  cause  the  processing  to  be  started  from  any 
particular  sample  on  the  record.  Essentially,  in  the  program  there  is 
a  pointer  to  the  next  data  sample  to  be  processed.  This  pointer 
has  three  parts:  the  file  number,  the  record  number,  and  the 
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position  on  this  record.  These  commands  change  one  or  more  elements 
of  the  pointer.  The  elements  can  be  changed  either  relative  to  the 
current  value  or  an  absolute  value  can  be  given  to  any  of  them. 

There  are  thus  six  commands  in  this  group. 

The  three  commands  SPFIL  I,  SPREC  I,  SPPOS  I  change  the 
specified  element  relative  to  the  current  value.  I,  which  is  the 
displacement,  can  be  positive,  negative,  or  zero.  The  other  three 
commands  FI I£H  I,  RECCH  I  and  POSCH  I  specify  the  absolute  value 
of  the  particular  element  of  the  pointer.  Here  I  can  only  be 
positive.  The  SPFIL  and  FII£H  commands  position  the  tape  to  the 
beginning  of  a  new  file  and  hence  alter  the  record  number  and  the 
position  also.  SPREC  and  RECCH  position  the  tape  to  the  beginning  of 
the  new  record  within  the  same  file.  Hence,  the  position  pointer 
could  be  altered.  For  the  SPPOS  and  the  POSCH  commands,  if  the 
new  position  falls  within  the  same  record,  the  record  number  does 
not  have  to  be  changed.  Otherwise,  the  tape  has  to  be  moved  forward 
or  backward,  depending  on  the  new  record  number. 

If  at  any  point  during  the  positioning  of  the  tape,  either  end 
of  either  the  volume  or  the  file  (in  case  of  record  and  position 
commands)  is  reached,  the  tape  stops  and  an  appropriate  message  is 
printed  out. 

(ii)  Commands  to  perform  the  actual  analysis:  The  analysis  programs 
exist  in  the  form  of  subroutines  which  are  called  by  the  main 
program.  These  subroutines  are  modules  which  are  overlaid  in  core. 
Thus,  a  new  module  can  easily  be  inserted  without  overrunning  the 
core.  The  programs  available  presently  are: 


18 


(a)  PLOT,  which  simply  plots  the  data  on  the  plotting  device. 

(b)  FIX  I  ,  which  determines  the  fixation  points  in  the  data 
according  to  an  algorithm  described  earlier  ant!  plots  and 
numbers  these  points.  I  is  a  parameter  for  the  algorithm.  An 
example  for  using  this  program  is  shown  in  Figures  III.l  and 
III. 2. 

(c)  VARNC  I,  which  determines  the  velocity  and  variance  of 
a  moving  window  of  data  and  plots  out  the  actual  data,  the 
velocity  and  the  variance.  I  is  the  length  of  the  window. 

An  example  is  shown  in  Figure  I I I. 3. 

For  instance,  in  the  prediction  scheme  to  be  described, 
we  set  a  certain  threshold  and  when  the  variance  crosses  this 
level,  the  predictive  process  starts.  A  certain  number  of  points 
(which  can  be  set  by  the  program)  after  the  onset  of  the  saccade 
are  read  in,  the  variance  calculated  and  a  curve  is  fitted  to 
this  initial  part.  This  gives  predicted  values  of  future 
variance.  Direction  information  is  obtained  from  this 
initial  part  of  the  saccade  and  combining  the  two,  v.e  get  the 
predicted  movement . 

(d)  WTDISP,  which  gives  a  simulated  three  dimensional  weighted 
display  of  the  eye-movement  data.  The  X-Y  plane  is  divided  into 
a  grid  and  the  "weight"  of  each  grid  point  is  plotted  along  the 
third  axis.  The  weight  is  obtained  by  taking  a  certain  portion 
of  the  data  and  then  for  each  sample  point,  assigning  some  weight 
to  the  corresponding  grid  point  and  to  points  around  it.  This 
is  based  on  the  fact  that  due  to  inaccuracies  in  the  instrument 
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! i KU re  III.l  A  Plot  ol  Eye-Movement  Data 


Figure  III. 2  Fixations  m  the  Above  Data 
Determined  by  the  FIX  Program 
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and  due  to  other  noise  sources,  a  given  sample  point  actually 
represents  a  probability  distribution  around  that  point.  At 
present,  a  normal  distribution  is  used.  An  example  is  given  in 
Figure  1 1 1. 4  Fixations  show  up  as  humps  in  the  figure. 

(iii)  Miscellaneous  commands:  These  commands  enable  us  to  filter 
the  data  with  a  three  pole  Butterworth  low  pass  filter  (FILTEh  i, 
i=l°f  ) >  to  resample  the  data  (COMP  i  will  use  every  ith  point)  to 
determine  the  present  position  of  the  tape  (LOCATE)  and  to  exit 
from  :he  program  (EXIT) . 

The  data  is  acutally  read  off  the  magnetic  tape  by  a  subroutine 
MTREAD  which  is  requested  by  other  programs  when  data  is  needed. 

By  using  the  switches  on  the  console,  it  is  possible  to  read  in  one 
point  at  a  time,  manually,  so  that  the  execution  of  the  ]  rograrns 
can  be  observed  more  easily.  It  is  also  possible  to  exit  from  any 
of  the  analysis  programs  by  means  of  a  switch. 

3 .  Methods  for  the  Prediction  of  Eye-Fixation  Points 

We  have  developed  two  predictive  processes.  At  present,  both  of 
these  methods  are  being  tried  out  and  showed  promising  results.  Refinements 
are  in  progress. 

(i)  The  Position-Variance  Method 

Taking  n  samples  (X  Y  ) ,  (X  ,Y  ),  ...  ,  (X  ,Y  )  of  the  eye- 

112  2  n  n 

movement  data,  we  calculate  the  means  by 


n  n 


n  n 
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f  igure  HI. 3  A  Plot  of  Eye  Movements, 
Sample  Mean  Velocity  and  Variance  l’rom 
the  VARNC  Program.  The  Peaks  in  the  Velocity 
and  Variance  Curves  indicate  Saccades. 


Figure  III. 4  A  Plot  of  Eye-Movement  Data  and 
a  Weighted  Display  by  the  WTDISP  Program.  The  Peaks  Indicate  Fixations. 
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and  the  variance  in  the  sample  window  by 


V  = 


77,2 


11  o  n 

E  <x,  -  x>“  +  E  (Yi  -  Y) 

i=l  i=l 


/n 


It  has  been  observed  that  the  variance  of  a  window  entirely 
in  the  fixation  mode  is  lower  than  a  certain  threshold  value.  When 
a  saccade  starts,  the  variance  rises,  reaches  a  peak  and  then  settles 
down  again  when  the  eye  fixates.  Thus,  by  setting  an  empirically 
determined  threshold,  the  onset  of  a  saccade  can  be  determined. 

After  this  point,  by  fitting  sinusoidal  curves  to  the  variance, 
we  can  predict  the  future  variance  values  which  along  with  the 
straight  line  assumption  enable  us  to  predict  the  next  fixation  point. 

(ii)  The  Velocity  Method 

In  the  above  method,  a  certain  time  lag  is  present  due  to  the 
width  of  the  window.  The  response  is  slow  because  of  the  averaging 
process  over  n  points.  To  overcome  this  difficulty,  we  used  the 
velocity  information  in  the  data.  By  using  the  eye-movement 
velocity  information,  we  developed  and  implemented  a  scheme  for 
prediction  of  a  new  eye  position  at  the  termination  of  a  saccadic 
movement  about  half  way  before  that  new  position  has  been  reached 
by  the  eye.  The  scheme  is  based  on  the  observation  that  the  eye- 
movement  velocity  is  very  nearly  symmetrical.  The  velocity  during 
a  saccade  increases  to  a  maximum  value,  then  decreases  back  to 
zero  in  approximately  the  same  manner  as  the  increase.  Thus,  by 
detecting  the  peak  value  of  the  velocity,  the  distance  the  eye  has 
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moved  since  the  beginning  of  the  saccade  up  to  the  time  of  this 
peak  can  be  doubled  to  get  a  prediction  of  the  eye-position  when 
the  velocity  will  decrease  to  zero  and  the  eye  has  arrived  at  a 
new  iixation  point.  Therefore,  when  a  saccade  is  only  half  completed, 
a  prediction  of  where  the  eye  will  be  at  the  end  of  the  saccade  can 
be  made. 

A  block  diagram  of  the  scheme  is  shown  in  Figure  III. 5.  X-Y 
eye-position  data  from  a  Biometrics  Eye-Movement  Monitor  is  sampled 
either  on-line  directly  from  a  subject  or  from  analog  tape.  Sampling 
rates  presently  used  are  1  millisecond  and  5  milliseconds.  The  data 
are  then  smoothed  by  using  a  digital  lilter  of  a  bandwidth  of 
approximately  0  to  20  Hz.  This  velocity  calculation  for  one  channel 
is  performed  as  follows: 

"velocity"  =  X(n)  -  X(n-l) 

where  X(n),  X(n-l)  are  two  successive  samples.  This  result  can  be 
divided  by  the  sampling  period  T,  iu  order  to  obtain  the  correct 
magnitude  for  the  actual  velocity.  However,  the  algorithm  only 
requires  a  value  proportional  to  the  velocity,  so  this  division 
is  unnecessary. 

During  a  fixation  this  velocity  fluctuates  about  zero  with  some 
variance,  which  is  determined  for  each  subject  in  each  experiment. 

A  threshold  detector  is  used  to  monitor  this  fluctuation.  When  the 
velocity  increases  monotonic al ly  above  the  maximum  fluctuation  level, 
it  is  decided  that  a  saccade  has  begun.  At  this  time,  the  on-going 
means  of  the  X-Y  positions,  which  have  been  calculated  throughout 
the  fixations,  are  saved  for  later  analysis.  The  peak  detector  now 
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begins  to  look  for  peaks  in  the  velocity  curve.  When  a  peak  is 
detected,  the  X  position  value  at  the  peak  time  is  used  in  the 
equation  below  to  predict  what  the  final  position  will  be  at  the 
end  of  the  saccade: 


final  position 


2  (X  -  X  .  .  )  +  X 

peak  initial  initial 


'X  -  X. 

peak  initial 


wh<  re  the  mean  value  ol  the  X  position  which  was  stored 

at  threshold  detection.  The  same  calculations  are  made  simultaneously 
to  predict  the  Y  position. 

As  an  example,  we  show  the  prediction  process  in  Figures  III. 6 
thiougl  IH.  Id.  In  each  iigure,  wc  show  the  eye  movement  on  an  X-Y 
plot,  the  horizontal  (X)  velocity  vs.  time,  and  the  horizontal  (X) 
eye  position  vs.  time.  In  Figure  III. 6,  we  show  the  onset  of  the 
saccade.  The  intermediate  eye-movement  processes  are  shown  in 
Figures  III. 7  and  III. 8.  In  Iigure  III. 9,  the  velocity  reaches 
its  peak  and  the  prediction  of  the  fixation  point  is  made.  Finally, 
in  Iigure  III. 10,  we  show  that  the  eyes  arrive  at  the  predicted 
fixation  point. 

The  method  just  described  is  simple  and  useful;  however,  it 
restricts  us  to  make  prediction  only  when  the  eye  reaches  the 
halfway  mark  of  its  journey.  We  are  in  the  process  of  improving 
this  scheme.  Various  methods  have  been  contemplated.  Before  we  try 
any  of  these  schemes,  we  think  a  thorough  analysis  of  saccades  is 


in  order. 


figure  I II. 7  Eye  Movement  during  a  Saccade 
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Figure  III. 9  The  Velocity  of  the  Saccade  having 
just  Reached  the  Peak  Value  and  the  Next 
Fixation  Point  Predicted 
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figure  III. 10  The  Saccade  Terminating  at 
the  Predicted  Fixation  Point 


It  has  been  hypothesized  that  the  eye  does  not  observe  anything 
during  the  saccadic  movement.  Thus,  before  the  saccade  is  made,  the 
decision  is  made  about  where  to  move  the  eye  and  with  what  velocity. 
Any  information  about  these  parameters  in  the  eye-movement  data 
should  then  be  present  in  the  data  immediately  preceeding  the  saccadic 
movement.  Thus,  by  analyzing  the  fixation  data  just  before  the  onset 
of  a  saccade,  we  can  determine  in  what  form  this  information  is 
present  and  whether  we  can  use  it  to  predict  the  onset.  Essentially, 
some  velocity  and  direction  analysis  is  performed  on  this  data. 

For  this  purpose,  the  onset  of  a  saccade  is  determined  by  the 
velocity  of  filtered  data.  When  this  velocity  crosses  a  certain 


threshold,  we  mark  the  end  of  the 


saccade.  Then  a  window  of 
data,  consisting  of  the  saccade  and  some  samples  before  and  after 
the  saccade,  is  taken.  These  -saccade  samples'  for  a  particular 
saccade  (e.g.,  from  point  PQ  to  are  collected  and  then  analyzed. 
In  addition  to  this,  the  actual  saccade  data  is  used  to  obtain  the 
characteristics  of  saccades  such  as  velocity,  length,  duration, 
variance,  etc.  Knowledge  of  these  characteristics  is  essential 
in  building  a  sound  predictive  model. 


Iv-  THE  "seer"  program  for  visual  scanpaths  study 

1  •  Introduction 

A  computer  program  is  desperately  needed  for  the  study  of  visual 
scanpaths.  We  believe  that  a  superior  scanpath  of  visual  inspection, 
i.e.,  one  that  results  in  a  superior  memory,  is  more  consistent  and  also 
less  probable  of  natural  occurrence  than  a  visual  scanpath  resulting  from 
an  inspection  strategy  that  is  less  consistent.  It  is  important  to 
determine  the  features  that  differentiate  scanpaths  associated  with 
superior  imagery  from  scanpaths  associated  with  inferior  imagery. 

In  this  section  we  descril e  version  1A  of  SEER,  a  PDP-15  program 
allows  on  line  analysis  of  visual  methods  of  object  recognition.  SEER 
permits  the  experimenter  to  display  previously  defined  images  on  a  Tektronix 
611  CRT.  As  a  subject  views  these  images,  data  may  be  taken  and  stored 
on  magnetic  tape  through  the  use  of  the  computer  and  a  Biometrics  Eye- 
Movement  Monitor  (Type  SG) . 

Here  we  describe  the  first  operational  version  of  SEER.  The  program 

has  been  used  in  its  present  form.  Further  modifications  and  additions 

are  in  order  to  make  SEER  more  versatile.  At  the  end  of  this  reporting 

period,  this  program  has  not  yet  been  applied  to  the  study  of  visual 

scanpaths,  since  it  is  barely  finished  We  nim  tn  *i  • 

y  iiiusnia,  we  plan  to  use  this  program  heavily 

in  the  immediate  future  to  study  the  various  scanpaths  in  relation  to 

visual  memory. 


2.  Description 

The  most  basic  SEER  entity  is  the  "picture."  A  picture  corresponds 
to  a  set  of  data  collected  together  and  given  a  symbolic  name.  Currently, 


all  picture  names  are  of  the  f 


orm 
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PCXXX 


where  XXX  is  a  non-zero  decimal  integer  less  than  999  (e.g.,  PC174, 

PC  Oil).  Notice  that  leading  zeros  must  be  specified  for  integer  values 
less  than  100. 

Ihe  data  composing  a  picture  consists  of  an  ordered  set  of  points 
which,  when  displayed  by  the  GS-15  Graphics  System,  produces  an  image 
on  the  surface  of  the  CRT.  Pictures,  i.e.,  groups  of  display  data,  may 
leside  in  three  different  locations  in  one  of  two  possible  formats 
(symbolic  and  binary). 

Pictures  in  symbolic  form  may  reside  on  disk  o^  DECtape  as  standard 
RSX  files.  Normally,  a  DECtape  contains  the  "library”  of  all  pictures 
available.  As  part  of  the  preparation  for  an  experiment,  the  user  would 
transfer  those  pictures  that  are  needed  to  the  disk  from  the  DECtape 
library.  Ihe  library  could,  of  course,  reside  on  the  disk  if  sufficient 
spice  were  available.  The  picture  files  on  disk  are  collectively  called 
the  picture  "album."  In  a  later  version  of  SEER  the  album  will  be  a 
single  random  access  binary  file  rather  than  a  group  of  symbolic  files. 
This  change  will  allow  faster  manipulation  of  pictures  since  binary  reads 

are,  in  general,  much  faster  than  symbolic  (e.g.,  Fortran  FORMATed 
READs) . 

Pictures  may  also  reside  in  the  "current  picture."  The  current 
picture  consists  of  a  "point"  vector  (actually  X  and  Y  vectors)  of  3500 
points  and  a  variable  specifying  how  many  points  are  in  the  current 
Picture.  Thus,  the  current  picture  is  located  in  the  computer's  main 
memory  as  a  collection  of  binary  integers.  Although  the  experimenter 
has  logical  access  to  a  number  of  pictures,  in  fact  he  may  only  work 
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with  one  at  a  time.  This  single  picture  that  is  of  current  interest 
is  the  "current  picture." 

As  explained  in  detail  in  Section  3,  there  are  numerous  SEER  commands 
that  work  with  or  on  the  current  picture.  For  example,  the  CREATE  sub¬ 
system  allows  the  user  to  "paint"  pictures  by  drawing  simple  geometric 
figures  (points,  lines,  rectangles,  circles,  etc.)  at  specified  locations 
on  the  CRT.  As  the  pictures  are  drawn  on  the  display,  the  points  defining 
these  pictures  arc  simultaneously  added  to  the  current  picture.  Once  the 
picture  has  been  "drawn"  it  may  be  saved  as  a  permanent  file  on  the  disk 
with  the  SAVE  command.  The  DELETE  command  is  also  available  for  erasing 
previously  drawn  figures  in  the  current  picture,  thus  providing  a  simple 
but  viseful  error-correcting  capability. 

Data  acquisition  is  accomplished  through  the  DRAW  sub-system.  In 
a  typical  experiment  the  FETCH  command  would  be  used  to  transfer  a  picture 
from  the  album  to  the  current  picture.  Once  the  current  picture  is  specified, 
it  may  be  examined  with  the  PREVIEW  command.  This  causes  the  entire 
picture  to  be  quickly  drawn  on  the  CRT. 

When  the  user  is  convinced  that  the  correct  picture  has  been 
FETCHed  (or  the  subject  has  had  a  chance  to  PREVIEW  the  picture),  he 
may  initiate  the  "taking"  of  eye-movement  data  by  using  the  VIEW  command. 

VIEW  will  cause  the  digitized  data  to  be  written  onto  9- track  magnetic 
tape  which  may  later  be  analyzed  off  line. 

A  major  reason  for  the  creation  of  SEER  was  the  need  for  a  simple 
sof twarc/hardware  system  that  could  determine  with  a  high  degree  of 
accuracy  where  on  the  CRT  a  subject  is  directing  his  visual  attention. 

The  proposed  solution  is  to  have  a  subject  trace  the  current  picture 
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,h°  SUbJOCt  “»>«*  »  ”0«t”  that  slowly  moves  over  every 

11,10  1,1  ,hC  **»“*>  "hll“  ltlolli toring  his  eye  movements.  Thus,  this 
traced  picture,  due  to  unavoidabie  noise  i„  the  system,  win  be  a  distorted 
version  ol  the  original  undistorted  picture. 

'OW,  if  the  subject  is  asked  to  look  freely  at  the  original  picture, 
the  data  values  correspond!,,,;  to  his  "look”  win  be  distorted.  However, 
fince  ...  have  ,he  entne  distorted  tmaBo  of  the  current  pteture  on  magnetic 
tape  (along  with  the  undtstorted  picture),  those  distorted  values  may  be 
Wd  back  into  locations  in  the  original  undistorted  picture  (assuming 
thf  mapping  is  wt  tit  ionary  over  the  ,  rnc  utg/view,  ng  And  tho 

'  ’*  ,'ct'0,',k'<l  L"  to, m,s  of  actual  Cltr  raster  addresses,  we 

may  finally  determine  the  actual  phystca,  location  that  the  subject  was  tiewing, 
a*  "  of  implement, ng  the  above  procedure,  t„e  current 

verstoh  ol  SEER  allows  data  ac„uisition  ojlj,  when  it  is  drawing  (tracing, 
on  the  CUT.  There  presently  is  no  facility  i„  take  eye-movement  data 
static  display.  Another  limitation  to  keep  in  mind  is  that  the 
digit izing  rate  is  directly  proper, rona,  the  rate  a,  which  a  picture 
is  traced  by  VIEW  (sec  the  SPEED  command  , he  nek,  section). 

-j  •  l  s  age 

A.  General 

The  SEEK  system  consists  of  a  resident  executive  and  a  number  of 
ove flayed  sub-systems  in  order  to  efficiently  utilise  core  memory 
modular  progiamming  extensions.  It,  general,  SEER  will 
prompt  the  user  (on  U1E-12)  for  a  command,  perform  the  regucsted 
function,  and  then  re-prompt.  The  SEER  executive  uses  "f  as  its 


prompt  while  ail  of  the  sub-systems  use 


When  SEER  is  initially 


started  ("KEQ  SEER"),  control  is  given  to  its  executive  which  writes 

SEER  VIA 

°h 

on  LUN-13  and  then  waits  for  an  executive  (major)  command  to  be 
entered  by  the  user. 

Th  Te  are  three  valid  syntatic  forms  of  SEER  commands.  The  first 
is  stmply  a  blank  line.  If  a  blank  (or  null)  line  is  input,  control 
is  returned  to  the  next  higher  command  level.  For  example,  if  a  blank 
line  is  input  to  a  sub-system,  SEER  will  return  to  the  major  command 

^  ^ '  If  a  blank  line  is  input  as  a  major  command,  SEER 

will  exit  to  the  RSX  system  (i  e  cirri,  + 

system  v i . e . ,  SEER  terminates  its  execution). 

The  second  command  form  is  a  string  of  characters  that  start 

isk  (  *  ).  This  command  is  a  "comment"  and  is  ignored 
by  SEER.  it  has  been  included  mainly  as  an  aid  to  documenting 
AUTOmatic  files.  The  AUTO  command  (to  be  included  in  a  future 
version)  allows  the  experimenter  to  build  a  file  of  SEER  commands 
that  are  input  automatically  to  the  system,  thus  freeing  the  user 
from  having  to  manually  input  commands  during  a  real  experiment. 

All  other  SEER  commands  are  of  the  following  form: 

\keyword) [^parameters)]  . 

As  a  means  to  save  command  input  time  without  forcing  a  cryptic 
command  language  on  the  user  (e.g.,  DDT,  QED,  ...),  SEER  requires 
the  user  to  type  only  as  many  characters  as  are  necessary  to  uniquely 
specify  a  keyword.  For  example,  to  invoke  the  DRAW  sub-system,  only 
a  ’D"  must  be  typed;  to  invoke  the  CREATE  sub-system,  both  the  "c"  and 
the  "R"  must  be  input  since  the  system  has  two  major  commands  beginning 
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with  '  C"  (CREATE  and  CALIBRATE). 


Onco  enough  characters  have  been  input  to  uniquely  specify  a 
command,  SEER  will  complete  the  rest  of  the  keyword  by  typing  the 
remaining  characters  in  it  (unless  not  in  VERBOSE  mode;  see  VERBOSE 
command).  In  addition,  if  the  command  requires  no  parameters,  SEER 
will  automatically  return  the  carriage.  A  SEER  parameter  list, 

parameters;  ,  is  a  list  of  one  or  more  integers  separated  by  either 
a  comma  or  blanks. 


B.  SEER  Major  Command; 


CALIBRATE 


Ihis  command  causes  the  execution  of  a  calibration  procedure. 
Currently,  the  calibration  consists  of  the  presentation  of  a  five 
point  matrix  on  the  CRT.  As  the  subject  fixates  on  the  calibration 
points,  SEER  records  the  eye-movement  data  on  magnetic  tape  as  in  the 
VIEW  command.  This  calibration  data  may  be  used  during  later  analysis 
to  correct  for  "zero  shift"  and  scaling  transformations. 


CREATE 

DRAW 

Both  of  these  commands  do  nothing  more  than  loading  and  passing 
control  to  their  respective  sub-systems.  Each  sub-system  has  numerous 
minor  commands  which  are  discussed  below  in  Sections  3.C  and  3.D. 


SPEED  N 


The  SPEED  command  determines  the  rate  at  which  VIEW  traces  the 


current  picture  (default  value  -  2048).  Earlier  in  this  manual, 
it  was  stated  that  the  digitizing  rate  is  directly  proportional 
to  SPEED.  Roughly  speaking,  a  SPEED  of  1000  gives  a  digitizing 
rate  of  approximately  200  Hz  (1  sample/5  ms).  As  in  the  GS-15 
Graphics  System,  "SPEED  999"  specifies  that  SPEED  should  be  dynamica 
taken  from  the  PDP-15  data  switches. 

VERBOSE 

Keyword  completion  is  controlled  by  this  command.  When  VERBOSE 
is  requested,  it  will  disable  (enable)  keyword  completion  if  current 
enabled  (disabled). 

C.  CREATE  Minor  Commands 

Ihe  CREATE  sub-system  is  used  to  "paint"  (i.e.,  define)  SEER 
pictures.  In  general,  pictures  are  composed  of  one  or  more  simple 
geometric  figures  such  as  circles,  lines,  etc.  All  CREATE  commands 
that  place  figures  into  the  current  picture  must  specify  where  on 
the  CRT  screen  the  figure  should  be  placed.  Locations  on  the  screen 
are  denoted  by  coordinate  pairs  (X,Y),  where  by  X  and  Y  are  non¬ 
negative  integers  less  than  1024.  As  shown  in  the  following  diagram, 
(°,0),  (0,1023),  (1023,0),  and  (1023,1023)  correspond  to  the  lower 
left,  upper  left,  lower  right,  and  upper  right  corners  of  the  CRT, 
respectively ; 

1023^ 

Y 

° - T 

0  X  1023 
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CIRCLE 

The  CIRCLE  command  will  add  a  circle  of  radius  K  and  center 
(X,Y)  to  the  current  picture.  It  is  possible  to  cause  an  "OVERFLOW" 
error  with  this  and  other  CREATE  commands.  This  occurs  whenever 
an  attempt  is  made  to  CREATE  a  picture  that  requires  more  than  3500 
points  for  its  definition. 

DELETE  [N] 

DELETE  will  remove  the  N  most  recently  entered  figures  from  the 
current  picture.  If  N  is  not  specif ied,  N  is  assumed  to  equal  one. 

LINE  X , Y , X ' , Y 1 

This  command  will  connect  the  points  (X,Y)  and  (X',Y')  with  a 
straight  line  and  add  the  line  to  the  current  picture. 

POINT  X,Y,N 

A  user  may  add  a  point  to  the  current  picture  with  the  POINT 
command.  Or,  if  necessary,  N  copies  of  the  point  (X,Y)  may  be  added 
by  specifying  N  appropriately. 

RECTANGLE  X,Y, WIDTH, HEIGHT 

A  rectangle  with  the  specified  width  and  height  is  added  to  the 
current  picture.  The  lower  left  corner  of  the  rectangle  will  be 
located  at  (X, Y) . 

SAVE  XXX 


The  SAVE  command  will  place  a  copy  of  the  current  Dicture  into 


the  album"  (a  collection  of  disk  resident  symbolic  files)  with  the 
name  "PC'XXX."  XXX  must  be  in  the  range  1-999.  If  a  picture  with  this 
name  is  already  in  the  album,  it  will  be  replaced  by  a  copy  of  the 
current  picture.  The  user  need  not  specify  leading  zeros  in  XXX 
(e.g.,  "SAVE  l'  will  save  the  current  picture  with  the  name  "PC007"). 

P.  DRAW  Minor  Commands 

IRASK 

The  ERASE  command  unconditionally  erases  the  CUT. 

KETCH  XXX 

Pictures  are  transferred  from  the  album  to  the  current  picture 
with  this  command.  XXX  specifies  that  the  picture  to  be  transferred 
is  "PCXXX"  (sec  the  SAVE  command).  The  image  o.i  the  CRT  is  not 
modified  in  any  way  by  KETCH. 

PREVIEW  [XXX] [,N] 

If  this  command  has  no  parameters,  SEER  erases  the  CRT  and  then 
draws  the  current  picture.  A  KETCH  is  first  made  to  picture  "PCXXX" 
whenever  the  first  parameter  is  present  (thus  making  "PCXXX"  current). 
If  N  is  specified,  the  current  picture  is  drawn  N  times.  Unlike 
VIEW,  PREVIEW  draws  the  current  picture  as  fast  as  possible  (i.e., 
independent  of  the  current  value  of  SPEED). 

VIEW  [XXX][,N] 

VIEW  is  identical  to  PREVIEW  except  that 

(1)  the  CRT  is  not  erased, 
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- ...  . . . . . . .  - 


(2)  the  rate  at  which  the  current  picture  is  drawn  is  dependent 
on  SPEED,  and 

(3)  digitized  data  is  taken  and  recorded  on  magnetic  tape, 

tor  example,  the  command  VIEW  (with  no  parameters)  will  draw 
the  current  picture  at  a  rate  determined  by  SPEED  and  record  the 
corresponding  eye-movement  data  on  magnetic  tape. 
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V.  EEG  SIGNAL  MODELING 


To  bettor  understand  the  relation  between  visual  stimuli  and  the  EEG 
signal,  we  have  used  a  nonlinear  oscillator  to  model  the  stimulus-response 
relationship.  In  particular,  we  are  studying  the  entrainment  of  the  alpha 
rhythm  by  periodic  photic  stimuli.  This  would  enhance  our  understanding 
of  the  eflect  on  EEG  signals  exerted  by  the  repetitive  photic  stimulation 
and  the  various  degrees  of  efficiency  of  the  stimulus  falling  on  various 
phases  of  the  EEG  signal.  There  is  every  reason  to  believe,  based  on 
the  results  of  other  researchers,  that  the  eye  takes  in  information  only 
when  it  is  fixating.  One  of  our  goals  is  to  guide  the  eye  to  fixate  only 
at  those  instances  corresponding  to  the  EEG  ph  es  while  the  brain  is  in 
the  best  state  for  processing  information.  We  relieve  that  there  are 
relations  between  the  effective  phases  for  stimulation  and  the  optimal 
phases  for  fixations.  It  is  hoped  that  the  development  of  a  model  for  the 
entrainment  process  will  elucidate  the  behavior  of  the  alpha  frequency  and 
phase,  and  thus  lead  to  a  more  practical  brain  state  estimator  and  predictor 
for  use  in  our  memory  studies.  Here,  we  present  the  nonlinear  oscillator 
model  and  some  preliminary  results, 

1  •  Model  for  Photic  Entrainment  of  EEG 

A  nonlinear  oscillator  known  as  the  Van  der  Pol  oscillator  has  been 
used  to  model  the  entrainment  behavior  of  the  EEG  alpha  rhythm.  Let  us 
denote  the  EEG  signal  by  x(t) .  We  may  write  the  oscillator  equation  as 

2 

x  +  pf(x,x,  +  =  Eg  sin  oj^t 
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where  represents  the  autonomous  or  unstimulated  alpha  frequency, 
is  thG  s^mulus  frequency,  Eg  is  the  amplitude  of  the  stimulus,  and  p 
and  f (x , x )  determine  the  nonlinear  characteristic.  The  coefficient  p  is 
called  the  coupling  coefficient.  We  are  interested  in  the  resonance 
phenonmenon  which  occurs  when  uj0  «  .  .  Letting 

wa  =  us  +  and  E  =  es/p  - 


we  obtain 


x  +  -  —  pf (x , x)  r  pE  sin  ^t  -  pxA 


We  seek  an  asymptotic  solution  of  the  form 


x  -  a  cos  4>  +  pv^a.uyt,  y  )  +  P^  (a,  ugt  ,4, )  +  ... 


where  +  -  '~st  +  4>.  In  the  resonance  case,  the  phase  difference  between 
the  forced  oscillation  and  the  stimulus  oscillation  will  be  related  to 
the  amplitude  and  frequency  of  oscillation.  We  define  a  and  y  as  solutions 
to  differential  equations  of  the  form! 

da  2 

dt  ~  H-A^(a,4>)  +  p  A^(a,4>)  +  ... 

dt  =  Ws  +  HB1(a,4))  +  p2B2(a,4>)  +  .  .  . 

where  the  A^a.tf))  and  B^a,^)  are  periodic  functions  of  4>  with  period  2n. 

As  a  first  approximation,  we  can  write  for  small  p, 
x  =  a  cos  (w  t  4-  ) 

~3T  =  ^Va)4>) 

d<t> 

—  =  pBi(a,4,). 

If  we  choose  f(x,x)  =  (x'  -  l)x,  the  original  equation  becomes 
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2.2 

x  u(x  -  l)x  +  w  x  =  (J.E  sin  w  t. 

u  s 

Following  the  analysis  of  Bogoliubov  and  Mitropolsky,  we  obtain 
da  gE  coscfr 

dt  2ui 

s 

2 

dfo  _  pE  sin  4)  +  pau>s  +  pa  A 

dt  ~  2aw  (1  -  M/2) 

Under  ideal  conditions  of  entrainment,  da/dt  =  d4>/dt  =  0.  This  predicts 
a  phase  angle  of  90  degrees,  which  is  quite  close  to  that  usually  observed 
in  real  data  (see  Figure  V.l).  Work  is  progressing  toward  a  direct  comparison 
of  the  frequency  ranges  of  entrainment  in  the  model  and  in  human  subjects. 

The  range  is  a  function  of  the  coupling  coefficient  p  and  of  the  stimulus 
amplitude  K  .  In  the  human,  the  range  appears  to  be  a  function  of  alertness 
and  of  the  stimulus  intensity.  Current  efforts  are  directed  at  quantifying 
the  entrainment  range  for  several  subjects  at  various  levels  of  alertness. 

This  data  will  be  used  to  refine  the  model  of  entrainment  presented  here. 


2 .  Method  of  Analysis  and  Preliminary  Results 

The  alpha  parameters  are  collected  in  the  following  manner.  KEG 
potentials  are  measured  between  the  occipital  region  and  the  yoked  left 
and  right  earlobes  with  the  ground  electrode  over  the  mastoid  process. 

Left  and  right  occipital  potentials  are  recorded  on  analog  tape  in  addition 
to  the  analysis  procedure  to  be  described. 

The  subject  is  stimulated  through  closed  eyelids  with  a  stroboscope 
for  50  seconds  on,  then  50  seconds  off.  The  frequency  of  stimulation  is 
typically  at  the  subject's  mean  alpha  frequency.  In  addition,  he  is 
requested  to  press  a  key  once  every  ten  seconds. 
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The  EEC  data  is  bandpass  filtered  ,1th  a  bandwidth  of  5  lit  and  a 
center  f regency  at  the  subject’s  mean  alpha  frequency.  The  parameters 
of  interest  are  computed  online  and  stored  digitally  „„  magnetic  tape 
in  addition  to  being  output  on  a  strip  chart  recorded  with  the  EEC. 

The  mean  alpha  period  is  determined  from  the  period  of  its  auto¬ 
correlation  function,  which  is  obtained  once  every  160  milliseconds  from 
a  digital  correlation  analyzer.  The  internal  integration  step  in  exponen¬ 
tially  weighted  with  a  time  constant  of  a  seconds  to  provide  a  reasonably 
stationary  period.  The  phase  between  the  alpha  rhythm  and  the  periodic 
stimulus  is  obtained  by  cross  correlation,  again  with  a  time  constant  of 
4  seconds.  The  correlational  analyses  provide  a  simple  real-time  evaluation 

of  two  key  parameters  and  minimize  the  computation  performed  by  the  computer 
running  the  experiment. 

we  show  some  results  of  this  analysis  in  Figure  V.l.  Note  the  changes 
in  the  alpha  period  and  phase  lag  as  a  function  of  the  stimulus  being  on  or 
off.  When  the  stimulus  is  on.  the  alpha  period  becomes  100  milliseconds, 
the  period  of  the  stroboscopic  stimulus.  When  the  stimulus  is  off,  the 
period  becomes  88  milliseconds,  the  autonomous  alpha  period  of  the  subject. 

In  addition,  the  phase  between  the  stimulus  and  the  alpha  rhythm  assumes  a 
fairly  stationary  value  of  90  degrees  when  the  stimulus  is  on  and  drifts 
at  no  particular  value  when  the  stimulus  is  off. 

This  data  clearly  exhibits  the  major  characteristics  of  the  non¬ 
linear  oscillator  model:  the  autonomous  frequency  u,„  is  entrained  to 
the  stimulus  frequency  uy  the  phase  lag  4,  assumes  a  preferred  value  of 
90  degrees:  and  the  autonomous  rhythm  returns  when  the  stimulus  is 


removed . 


msec 


Stimulus  on/off 


Turning  to  Figure  V.2,  we  note  that  the  mean  and  variance  of  the 
interresponse  time  for  key  taps  is  much  higher  than  in  Figure  V.l. 
Correspondingly,  the  variance  in  the  alpha  period  is  increased,  as  is  the 
nonstationarity  of  the  phase  lag  during  stimulus-on  periods.  These  relation¬ 
ships  lead  us  to  postulate  that  high  variance  of  the  alpha  period  and  the 
phase  lag  are  indications  of  drowsiness.  Similar  studies  have  shown  that 
these  characteristics  occur  in  the  nonlinear  oscillator  model  at  the  fringes 
of  entrainment.  Since  the  coupling  coefficient  (i  controls  the  range  of 
entrainment,  we  believe  that  |i  is  related  to  alertness.  This  would  suggest 
that  an  internal  "neuronal  clock"  is  reduced  with  drowsiness.  The  entrain¬ 
ment  of  the  alpha  rhythm  by  an  external  oscillation  is  seen  to  be  remarkably 
similar  to  the  behavior  of  the  nonlinear  oscillator  model  when  coupled 
to  an  external  oscillation.  Initial  investigations  indicate  that  the 
alpha  rhythm  will  entrain  to  frequencies  within  a  range  of  +  2  Hz  from 


the  autonomous  alpha  frequency. 


Entrainment  of  the  Alpha  Rhythm  at  Low  Alertness  U?vel 


100  sec.  150  sec 

,PA55  CHART 


VI.  CONCLUSION 

As  described  in  this  report,  our  work  has  been  proceeding  towards 
the  goals  that  we  proposed.  We  have  now  better  techniques  to  deal  with 
the  errors  and  distortions  of  the  eye-movement  data  as  measured  by  the 
Biometric  Eye-Movement  Monitor  unit.  We  have  implemented  some  pre¬ 
diction  schemes  which  predict  the  eye-fixation  points.  A  large  amount 
of  eye-movement  data  has  been  collected  and  is  being  analyzed.  The 
analysis  will  give  us  a  better  understanding  of  the  characteristics  of 
the  saccadic  movement  and  the  fixation.  It  is  expected  to  lead  to 
better  practical  predictive  models.  We  have  completed  computer  software 
dubbed  "SEER"  which  is  designed  for  the  study  of  visual  scanpaths  in  a 
new  way.  In  the  offing,  we  shall  see  new  interesting  results 
in  the  scanpath  study  with  the  aim  to  find  the  scanpath  which  will  result 
in  superior  memory.  An  EEC  signal  model  has  been  conceived  and  studied 
with  the  hope  that  it  will  aid  us  in  building  better  tracking  models  for 
the  brain  states.  We  have  also,  in  this  period,  expanded  our  computer 
systems  to  better  suit  our  needs. 

With  the  continuously  up-dated  information  concerning  the  eye 
position  and  brain  state  for  adjusting  the  stimulus  parameters  and  the 
tracking  and  prediction  techniques,  we  should  be  able  to  guide  the  eyes 
to  fixate  at  the  optimal  locations  and  the  optimal  time  instants  such  that 
the  vividness  and  persistence  of  the  desired  after-images  will  be  enhanced 
Future  works  are  planned  along  the  lines  to  get  a  better  handle  of  these 
optimal  locations  and  optimal  time  instants.  Armed  with  this  information, 
we  shall  attempt  to  devise  real-time  strategies  for  the  control  of  visual 
target  impression  and  visual  image  (memory)  persistence  and/or  recall 
ability. 
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